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Abstract: The field of visual communication is currently facing 
a period of transition to digital media. Two tracks of development 
can be distinguished; one utilitarian, the other artistic in scope. 
The first area of application pragmatically substitutes existing 
media with computerized simulations and adheres to standards 
that have evolved out of common practice. The artistic direction 
experimentally investigates new features inherent in computer 
imaging devices and exploits them for the creation of meaningful 
expression. The intersection of these two apparently divergent 
approaches is the structural foundation of all digital media. My 
focus will be on artistic explorations, since they more clearly 
reveal the relevant properties. The metaphor of a data particle 
is introduced to explain the general universality of digital media. 
The structural components of the data particle are analysed in 
terms of duality, discreteness and abstraction. These properties 
are found to have significant impact on the production process of 
visual information: digital media encourage interactive use, 
integrate, different classes of symbolic representation, and 
facilitate intelligent image generation. 

Keywords: computerized media, computer graphics, computer 
art, digital aesthetics, symbolic representation. 


Draft of an invited paper for VISUAL COMPUTER 
The International Journal of Comuter Graphics 
to be published summer 1986. 


digital media 0 





0. INTRODUCTION 

Within the last two decades we have witnessed a substantial increase in both quantity and 
quality of computer generated imagery. The work of a small community of pioneering engineers, 
scientists and artists started in a few scattered laboratories 20 years ago but today reaches a 
prime time television audience and plays an integral part in the ongoing diffusion of computer 
technology into offices and homes. The computer has rapidly evolved from a fancy calculator into a 
communication medium, a medium that, although able to encompass ail other existing media, is 
quite unique in structure and functionality. In my opinion, neither the underpinnings of this 
change nor its ramifications on our communication behavior and systems have been researched 
with the same enthusiasm or funding that characterizes the restless investigations and 
experiments of the technical aspects of this enterprise. 


It should not come as a surprise then, that we only know how to speak of the phenomena 
introduced with computer technology in technical terms. Questions that tackle the difficult 
problem of how the computer generates visual communiqués that are meaningful to a human 
perceiver are hardly asked and barely understood. What is the digital material made of and how can 
it be crafted into visual form? This preliminary study aims at establishing the notion of a deep 
syntactic-semantic structure of computer graphics. Much more work needs to be done before a 
coherent terminology and methodology will emerge at this intersection of communication, 
cognitive, and computer science and linguistics/semiotics which | suggest naming: digital 
aesthetics. 


Within the scope of this article | had to exclude the analysis of the computer as a “time 
entity" (Steve Wilson) vs. the classic techniques of still-frame animation, 


environmental/interactive installations, and media networks, even though | consider these 
manifestations integral to the digital medium. 


1. ON ART AND ARTIFACTS 

Initially, any computer generated picture could easily be spotted, due to the restricted 
means by which it had been generated. For some time the so called "computer look" with blocky 
pixels and flashy color grids became so trendy that it was copied in traditional graphic design. Ina 
second phase, many different art-styles were simulated on graphics systems with varying degrees 
of sophistication: constructivism, op and pop art, painterly abstractionism, comic strips and 
caricatures, oriental pen drawings, wood cuts and photographic realism. This attitude of mutual 
imitation belongs to the transitory times of the birth of a new medium and will soon lose its 
attraction. With the exception of photographic realism many examples of these styles have been 
amply published in magazines devoted exclusively to personal computers. The full spectrum of 
possible artistic approaches has even been demonstrated on small video game machines with no 
more than 12K Byte resident memory. (Dietrich, Molnar 1981). 


Even though realism of the quality that makes the computer generated image 
indistinguishable from a photograph remains an ambitious goal, researchers are confident they 
will accomplish such imaging fidelity. The surface quality or "look" of the resulting “realistic” 
image serves as a criterion for the technical achievement embodied in the programming and 
display of such a marvel of computer graphics. The likelihood of such a technical achievement 
forces us to completely disregard the visual output for an evaluation of the unique aesthetic 
properties of digital media, because they exhibit the same visual appearance like conventional 
media. 


The once dominantly visible traces of the structural components inherent in computer 
imaging technology are extinguished by affluent programming techniques that will completely 
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tame the previous overwhelming combinatorial abundancy. The ever increasing level of 
algorithmic problem solving is matched by a display resolution that is conceivably equivalent or 
higher than other visual media and supported by significant jumps in processing power and speed 
as well. Therefore the emphasis of critics will shift from a judgement on technical or conceptual 
achievement to artistic virtue. Such an evaluation can not refer in particular to computer art, 
but to art in general. A piece of computer assisted art will be confronted with the same questions 
as any other work of art: How does it respond and contribute to contemporary society, and how 
does it reflect and challenge human needs and desires ? 


It is not my goal to attempt to answer these questions here, neither to argue about the beauty 
of any of the pieces discussed. To gain some more freedom from the demanding connotation the 
term art invokes, | prefer to use the more generic word artifact to describe a computer 
generated entity. Artifact additionally emphasizes both its artificial origin and its physical nature 
in the real world: it is an artificially created fact. 


2. REPRODUCTION IN MASS MEDIA 

The invention of photography began a new age for visual communication, the age of technical 
reproduction. The rapid succession of new visual media like photography, film, television and 
video created a phenomenon similar to the literate society produced by print on an advanced 
plateau. The print medium, not only makes linguistic symbols storable like any handwritten text, 
but more importantly, reproducable in theoretically unlimited quantities. Its capacity for 
reproduction made printed media the dominating communication tools for about five hundred 
years, a historical time span that McLuhan christened "The Gutenberg-Galaxy". (McLuhan 
1962) 


The advancement of the television medium rests mainly on the immediacy of the electronical 
reproduction process. Images could be transmitted Jive anywhere in the world. Understandably, 
this power for mass communication became instrumental in ideological battles and advertising 
campaigns. The term Consciousness Industry (Enzensberger 1962) reflects the overall 
purpose of broadcasting and other forms of mass communication to function as tools to create social 
consensus by showcasing values and patterns of behavior intended to be internalized by the 
audience. 


In general, media such as photography, film and television had their impact as media of 
reproduction and supported the growth of mass communication. Computer generated information 
products can be absorbed by traditional mass media if they are produced adhering to established 
formats of distribution. On the other hand it will be shown that the uniqueness of computer 
imaging is most effective in the production sphere. During the production cycle digital media can 
both incorporate and simulate all other media as well as execute imaging concepts without any 
precedence. 


3. THE HEART OF THE MATTER: DATA PARTICLES 

Information is always comprised of two components: the representational sign that stands 
for something else and the physical carrier on which the sign is transmitted or stored. "Sign" has 
been defined by Charles Peirce as "something which stands to somebody for something in some 
respects or capacity" (Peirce 1897). The sign can only exist as a parasite of the carrier medium. 
It is the carrier that makes the transmission of internal mental activities communicable via 
externalization onto a physically perceptable entity. Much as the sign is bound to its carrier, its 
_meaning totally transcends the physicality of the signal's medium. Similarily a word conveys 
meaning through a social convention that defines a relationship between carrier, symbol and 
meaning in a completely arbitrary fashion. 
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All media operate on the basis of physical matter that is determined in one particular form 
and as such unchangable. The typeset letter, the modulated airwave, the photographic negative, all 
are confined by their materiality to be one thing and not another. The same then is true for the 
symbolic representation coupled to these media and transmitted by their virtue: each sign only has 
one fixed form of existence that points to its specific meaning or set of meanings. 


The ability to interpret signs and to reveal their meaning has been has been an exclusive 
human prerogative, until two of the founding fathers of Artificial Intelligence, Allen Newell and 
Herbert Simon, proclaimed the computer to be a symbol system, too: "Symbol systems are 
collections of patterns and processes, the latter being capable of producing, destroying, and 
modifying the former. The most important properties of patterns is that they can designate 
objects, processes, or other patterns, and that when they designate processes, they can be 
interpreted. Interpretation means carrying out the designated process. The two most significant 
classes of symbol systems with which we are acquainted are human beings and computers." 
(Newell, Simon 1975) 


Being a symbol system enables the human organism to express mental states and to 
communicate them, or conversely, to perceive and understand someone elses externalized states. 
Furthermore, symbols are the cognitive tools to internalize and comprehend with methods of 
abstraction a reality which exists outside and independently of the human organism. Language 
assumes this role to a point that it creates reality. "[This means] above all that language 
contains a definite Weltanschauung, that it determines the way we perceive and grasp reality. 
Hence, in this sense of a perceptual perspective, language creates our image of reality and imposes 
this image upon us. It is, as it were, a mold which brings order into the original chaos of reality 
‘in itself.” (Schaff 1973) 


It is questionable how these features of natural language compare to the formalism of digital 
representations. If we want to understand how the computer functions as a symbol system we will 
first need a better description of the material digital information is made of, and then we can 
proceed to categorize the formative processes invoived in manipulating the material into cognitive 
assemblies of signs. We will assert that the constituent informational unit in digital systems has a 
dual structure, is a discrete element, and is abstract to the degree of beeing void of any physical 
content. We will examine how each of these features contributes to the productive imaging 
potential of computer graphics. 


3.1. Duality 

The matter on which digital information resides however, is significantly different from any 
other material that can serve as a carrier medium. To grasp its essence | would like to introduce 
the metaphor of the data particle as the smallest undividable physical unit capable of carrying 
digital information. The binary properties of the data particle have to be understood beyond their 
meaning in a number system, where each number simply represents a given quantity. In this 
context the decimal or binary representation of a number is completely insignificant to the 
abstract value of the actual number it represents. | suggest conceiving of the binary nature of the 
data particle in an ontologicsrather than a numeric way. Ontologically speaking the data particle 


encompasses a dichotomy in‘one single entity: at the same time it contains its identity as well as 
its inverse. | 


The very substance of the data particle is dynamic. The current state a data particle might be 
in exhibits a twofold extension just like a coin has two faces. But both faces of the coin represent 
the very same value, whereas the data particle's states are contrasting each other. The activated 
one is identical with its current state. Concurrently the data particle carries the full potential to 
change this state to its opposite. The implications of this dual nature of the data particle are far 
reaching on every level of constructing conglomerates of data patterns that in turn serve as 


digital media 3 





carriers or codes for signs. The dynamics inherent in the duality of the data particle are directly 
coupled with its ever present potential to dynamically change the symbolic content encoded in a 
stream of data, regardless whether this change is effected by a human intervention or an 
algorithmic operation. 


3.1.1.Dynamics of Interaction 

The obvious implication of such dynamic nature is the potential both for dynamic change of 
an image and the interactive use of computers. In the early days of computer graphics interactive 
applications of computers were severely restricted by a lack of peripherals or tools that could 
mediate between the outside world, the user, and the computer. (Franke 1971/1985 and Dietrich 
1985) Artists were facing the computer as a self-contained system and resorted to the use of 
random number generators to introduce variation into their designs as they had to be totally 
conceptualized beforehand. Random numbers enabled them to realize a fraction of the dynamic 
capacities of the computer by altering the data parameters of a program in execution. The 
substitution of random number generation for creative interaction nevertheless falls short in at 
least one prominent aspect: random numbers can never exhibit the cognitive decision making 
quality a designer brings to bear in the creative process of aggregating symbols into a meaningful 
complex of signs. This is the case even if random numbers are strictly constrained by the 
programmer in a pre-forming stage. 


David Em became one of the first artists working with a now common set of interactive tools 
consisting of frame buffer and tablet (light-pen, mouse), a paint program and 3 dimensional 
rendering software. His imagination and interest were sparked right away when he saw a prototype 
of this interactive system configuration in a laboratory. The abstract design logic required to 
program computers before the advent of interactive systems would have not been attractive to 
him, because he regards immediate interaction as an essential outlet of his creativity: 


"The designer makes thousand upon thousands of decisions in generating a successful image, 
many of which do not even occur on a conscious level. Decisions come about as a result of 
experimentation, trial and error, mistakes, and occasionally from inspirations from far left field. 
There is no formular for why one thing works and not another, it is only through spontaneity that 
the creative flow happens. The frame buffer is the tool that makes this possible." (David Em 
1983) | 

Em identifies the frame buffer as the piece of hardware that can not only display the data 
patterns stored in the computer like a plotter can, but can make the change they are subjected to 
by the designer visible. Plotters trace fixed marks on paper, whereas the frame buffer 
illuminates light in accordance to a reserved portion of the memory that contains the image in 
binary format, the bitmap. Connections to and from the outside world are installed by input and 
output mapping into dataspace.’These mappings maintain an isomorphic relationship between the 
analog continuum of the position of the pen on a plane to the discrete image memory, and 
reversely, from the bitmap to the electronic display tube. Both conversions make the data 
particles accessible and give the operator full control over each one's particular state, including 
the option to change any particle to its inverse at will. The conversational cycle involving the 
visual dialog between human and machine is complete. The successful structural coupling 
(Manturana 1970) involves visually intuitive machine receptors and effectors and rests on the 
dual character of the data particle. Its current status can be dynamically switched by user actions 
that are mapped directly onto a chosen particle and the new value is immediately shown in the 
form of electronic light. 
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3.1.2. Algorithmic Growth | | 

We have seen how the dual character of the data particle opens up a full array of dynamic 
interaction for the designer controlling the process of affecting change to the data set that 
comprises the image. The same principle of change can also be implemented by algorithmic 
procedures without human intervention at all. The most fascinating examples are those that do not 
pre-determine the course of events but describe the inital data and procedural rules from which 
the computer autonomously generates more data that eventually encodes the rendering parameters 
of the final image. 


Alvy Ray Smith applied methods of data amplification to graphs that contain a given data 
string as well as a grammar establishing relationships between the data particles. New data is 
generated by recursive extrapolation on the existing set that yields more data exhibiting the same 
grammatical structure. (Smith 1984) This process can be interpreted such that initially a data 
space without limits exists wherein each data particle's value is still undefined or virtual. By 
executing the generator the data particles will be set according to the transformational grammar. 
After termination of the program the previously unstructured data space has become a set of 
well-formed data patterns. Smith used this approach to instruct the computer "to grow plants” on 
the display. After having generated the data patterns he interpreted them with a conventional 
rendering program. (*** Illustration # 9b ***) 


A similar project has been conducted for the last ten years by the Japanese artist Yoichiro 
Kawaguchi, who set out to model organic forms such as shells, tusks, and tendrils. The project that 
started with simple two dimensional line drawings has evolved over the years to render some of 
the most amazing images generated with computers. Kawaguchi himself summarizes the driving 
quality of the project in terms of self-propagating energy: 


"In the process of reaching this ‘Growth Il: Morphogenesis’, the very laws themselves 
concerning form have arisen and grown, creating their own present system which is 
self-propagating. In other words, the process has hypothesized something which retains energy 
within. It is something which has advanced one step beyond a simulation of a cross-section of the 
natural world. It is an approach to nature in another sense." (Kawaguchi 1985 b) 


Kawaguchi singled out the spiral as the geometric unit that constitutes the organic shape of 
many natural objects. After a brief period of developing the mathematics of the two dimensional 
spiral Kawaguchi used a cylindrical polyhydron as the three dimensional primitive to render his 
images. The algorithm provided him with the computation of the growth ratio between one unit and 
the next, as well as with a mechanism to recursively construct branches that were self-similar to 
the original formation. A small set of parameters to define the dimensions of the basic trunk 
cylinder, the relationship of height and diameter between parent and sibling units, and the 
branching angles at the joints, were the only data that fed the process of creating subsequently the 
growth of the object. (Kawaguchi 1982) 


Kawaguchi's work simulates the process of growth rather than modeling only one singular 
object by explicit definition. His algorithm derives its generative power mainly from its 
recursive structure that enables it to create new child branches by extracting harmonic ratios of 
data from the parent. The growth algorithm becomes the agent forming clusters of data particles, 
neither randomly nor by human intervention, but by dynamically replicating the morphology of 
the initial values embedded in a set of production rules. 


Kawaguchi's work received another dramatic growth spurt when he teamed up with Koichi 
Omura of Osaka University, the designer of a multi-processor graphics system, LINKS-1. This 
system has been optimized to model ray-traced scenes rendered on the basis of meta-ball 
primitives. These metaballs are the generalization of Jim Blinn's blobby molecules, a 
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technique to compute surfaces according to the density distribution of spheres or ellipsoids in 
space. (Blinn 1982) 


The property of meta-bails to visually melt into each other matches the simulation of 
organic growth especially well. Kawaguchi further enhances his recent imagery with advanced 
rendering techniques that include multiple transparencies and multiple texture-mapping 
(Kawaguchi 1985 a). The sheer amount of images he has produced over a few years is proof of the 
productive potential implemented in the growth algorithm. More importantly however, this 
project has been successful in realizing a genuine act of creation. From the simulation of aspects of 
natural phenomena it qualitively grew to create artifacts that offer a glimpse into the new visual 
territory So many set out to discover with the helo of computers. 


3.2. Discreteness 

The second property of the data particle is its discreteness. Any image is completely 
atomized into a group of separate data units, that visually can be interpreted as independent, but 
equivalent picture elements (pixels). Their size is identical and they are uniformly distributed 
into a raster grid across the picture plane. There are several repercussions to the discreteness of 
the data particle. Woody Vasulka sees here the underlying basis "to unleash some attack against the 
tradition of imaging, which | see mostly as camera-obscura-bound, or as 
pinhole-organizing-principle-defined. This tradition has shaped our visual perception, not only 
through the camera obscura, but it's been reinforced through the cinema and through television." 
Vasulka calls for "A Syntax for Binary Images" (Hager 1978) that is based on the dynamics of the 
discrete pixel derived out of the full set of arithmetic logical operations, rather than the holistic 
frame generated by an optical projection. As a matter of fact, the discreteness of the data particle 
has certain disadvantages for holistic types of imaging. Discrete elements can not form a visual 
continuum or to phrase it in technical terms: a raster display will always exhibit disturbing 
aliasing phenomena. A special branch of computer graphics deals with filtering and interpolation 
techniques to remedy this limiting display artifact by creating the illusion of continuity between 
adjacent pixels (anti-aliasing). 


But the discreteness of the data particle is the reason that faithful replicas can be made of 
digital information. Each data particle that is copied contains exactly the same information as the 
source. As a matter of fact, the binary value of the source and the copy is completely identical and 
can not be distinguished. Digital copying comprises a one-to-one mapping from one binary unit to 
another discrete particle without the imperfections and noise that enter a copy manufactured by 
analog methods of reproduction. In the analog process errors eventually accumulate to a point 
where an image that has undergone several generations of copying shows a noticable degradation 
from the original. In comparison, a digital copy remains identical to its source no matter how 
many times it has been copied. 


The notion of the original, already shattered by photographic and other technical 
reproduction techniques, has become totally obsolete in the digital domain. (See Walter Benjamin, 
“The Work of Art in the Ade of its Technical Reproduction", 1936) At least this is the case for the 
image in its digital format, An analog conversion in any form of (moving or still) hardcopy 
transfers this information into another realm, whose own laws will now govern the future status 
of the image. Since the artmarket values a piece of art in terms of the scarcety and uniqueness of a 
commoaditiy, it views the technological potential for limitless copies that can be put out exactly 
and easily as very problematic. 


3.2.1. Copying Made Productive 

We are concerned here with the productive potential of the digital copy and how it can be 
utilized during the formative stages of creating the image in the first place. If it sounds at first as 
a contradiction in terms to relate copying to creation, we will soon see digital replication as an 
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advanced technical instrument in the hands of a creative artist, a tool that has significant impact 
on the production process with computers. David Em describes those portions of his interaction 
with the computer, which are due to precise and efficient copying, as creative branching. He 
sheds new light on common computer terminology such as storage and retrievaf. 


"The ability to store and retrieve hundreds of images allow the artist to pursue an idea 
without ever losing the original source image. After reaching a fork in the road, the artist can 
choose one, and, by saving the current stage of the work, return to it later and pursue the other, 
which may dead end or turn into yet another fork. This is quite different from painting, where each 
stroke is a commitment from which there is no turning back more like a one way street with no 
turnoffs.” (David Em 1983) 


Em could have continued by turning to the more subtle copying that has direct implications 
for the structure of the resulting image. The so cailed paint brush for instance, is nothing more 
than a data pattern that is used as a template to be copied into an interactively specified location. A 
more complex version of this process takes place when symmetry operations are applied to the 
template by inverting the order in which the pattern is copied. Copying can even include 
processing when it is performed under conditional comparisons with the existing content of the 
picture area that is covered by the brush. In such case the contents of both the picture and the 
brush are read and combined by rules of arithmetic logic. This operation yields an astonishing 
spectrum of new pictorial variations based on the two previously existing data sets. 


Another processed variation of copying takes place when Em uses the image he painted into 
the two dimensional frame buffer merely as a reference map to be textured onto three dimensional 
objects. In this respect texture mapping is a copying process with an additional geometric 
dimension, that projects the data of a planar image onto three dimensional objects of arbitrary 
shape. In Em's repertoire the texture mapping techniques are indispensible for giving his spatial 
compositions a tangible complexity. Through his collaboration with Jim Blinn, Em was fortunate 
enough to be able to use some of the most sophisticated texture mapping techniques, known as bump 
mapping to simulate wrinkled surfaces, while they were still under development in the 
researcher's laboratory (Blinn 1978). Such software tools together with the logistic support Em 
received, formed an outstanding environemt for an individual artist. Further analysis of Em's 
work would reveal the innovative use he made of his resources; his achievement is to have made 
visual sense of a state-of-the-art graphics system. 


3.2.2. Components of a Pixel 

So far our description of the discrete character of the data particle assumed the simplest 
case of mapping one bit of data into one pixel. In many contempory frame buffers each pixel is 
actually represented by larger groups of bits. This additional data encodes additional features of the 
image;i.e. color, depth, transparency. This encoding scheme makes tt possible to either isolate one 
pixel out of all neighboring ones in the image plane and allows for the precise treatment of minute 
areas of the image. Or the multiple layers of representation of all pixels can be broken down into 
their individual components which opens access to each of the encoded features. 


The discrete form in which the data particles encode information becomes the very reason 
for the ability to classify image properties and perform activities on visually relevant groups of 
data. Among the more common techniques derived from this dialectic principle we find two types of 
color manipulation. The two differ depending on whether color results as a mixture out of directly 
encoded quantities of the primary colors red, green, and blue or via indexing into a separate color 
map. The former technique requires processing of the bitmap to increase or decrease particular 
values or compare values with arbitrarily set threshholds, etc. The second technique affects color 
change just by resetting values in the color map which are interpreted for all pixels with the same 
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value. 


Hirochi Iguchi utilized color map manipulation rather effectively in his piece "Amida 
Buddha". This image is based on a photograph of an ancient Japanese sculpture depicting the 
chanting monk Kuya. The little pipes flowing out of his mouth attempt to visualize the temporal 
succession of the chant's sounds. Iguchi used a video camera to digitize the photo, combined it with 
a "painted" background, and then restricted his aesthetic intervention to the application of an 
unusual color map. He selects the grey values that comprise a part of the head of the monk and sets 
them to widely contrasting colors. After this color change the originally smoothly half-toned 
contours of the head become sharply segmented and take on a meaning beyond simply rendering the 
surface. It is as if we can see the mental state of the monk. Since the pipes share some of the color 
values found in the head, they too show the same color patching. Thus Iguchi relates the external 
utterances of the chant with the mental contemplation of the monk and gives a completely new 
meaning to a picture well known in Japan. (“**see illustration #4 ***) 


While Iguchi utilizes the discrete nature of the digital image to exert control over groups of 
colors, | have been applying the same principle to organize a multi-bit frame buffer into several 
groups of images. In my own work | have been experimenting with methods that treat each bitplane 
of a frame buffer as a separate image entity. While masking off seven planes, | digitized a portrait 
into the remaining active plane. Successively | took eight shots, each into a different plane, and 
later combined all exposures into one single image by simply displaying all bitplanes at once. This 
technique can simulate the photographic multiple-exposure, but whereas the photograph always 
maintains its integrity as one holistic image, the digital version allows for the separation of each 
exposure at any time. Repeating only one of the exposures is easy in the digital domain due to the 
discreteness of the data, but the photographic exposures all share the same physical continuum and 
thus are not separatable. The major reason to do multiple exposure was my wish to have separate 
control over a range of digitizing parameters. Some exposures were taken with the lens adjusted 
slightly out of focus, while others were done with a minor change of the zoom. | selected these 
optical variations, because | had a composite image in mind, that no longer would resemble a 
photographical projection, but would take on the appearance of a free-hand drawing with 
expressive color strokes. (*** see illustration #5 ***) 


3.3. Abstraction and Representation 

Another aspect of the data particle is its absolute abstraction from physicality. It has logical 
value but not a physical dimension like pigment or a waveform. An image made up of pigments will 
forever be tied to that material in the same way a word is inscribed by the ink that prints the 
letter on paper. Both, image and word are members of different referential domains, and can not be 
brought together without violating the material by cutting and pasting. Digital data stands only for 
an encoded symbol regardless of whether it exhibits linguistic, numeric or visual meanings. 


lf we reveal the contents of a computer, many different layers of codes that are 
hierarchically related to egch other emerge so that one layer represents another one in operation 
beneath it. From any symbolic level, let's say, an image, we could be cascading down through its 
pixel description, the geometric data it contained before projection onto the i image plane, various 
layers of software, from highslevel languages in their source and compiled versions, to assembly 
and machine language routines. (Flores, Winograd 1985) On any level we can find a complete 
description of the status of the machine that in turn can be translated into any other description. 


Again, the consequences of the abstract character of the data particle are significant for 
imaging: images can be mixed and merged regardless of their origin once they take on digital 
format. Moreover, linguistic representations can easily be converted from language into numeric 
code as well as into geometric description that can be rasterized into pixel data. This information 
can be treated as visual information once it is encoded as geometric or pixel data pattern and can 
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then be collaged just like any other visual data. 


3.3.1. This is not an APPLE 

Hubert Hohn is one of the few artists to choose the computer not just as a tool but as the 
subject of his work. . Hohn expresses his stance towards computers as an image making device as 
an attempt "fo locate ourselves within the nature of digital processes and see what we become." 

"The digital computer is a symbol processor. 256 discrete patterns of signals are available 
for definition and manipulation as symbols. Everything here consists of symbols created through 
the tranformation of symbols by yet other symbols. There are no original thingy sort of things. 
The territory consists entirely of maps which refer to no territory but to other maps." 

"What | do is on floppy disks, and what the computer does is on paper printouts . My work 
exists only as binary magnetic code, and means nothing except during its execution, which is done 
by the computer. ... Multiple titles are used to encourage the viewer to be conscious of the work in 
a variety of ways, thus allowing it to exist in more than one state. My intent is to enliven the 
concepts while being fully consistent with the discipline of computing, for a world of multistate 
hardware and multireferent symbols is nothing if not a world of optional meanings.” (Hohn 
1985) 


One of Hohn's pieces, "Binary Data Dump", can serve as an excellent illustration of the 
layers of representation inherent in the machine as discussed above. The piece consists of a framed 
floppy disc, that contains the program, a list of titles, and a printout of all memory locations 
totalling 256 pages with 256 bytes of data each. Each byte is grouped to eight bits and each bit is 
represented by a "O" ora "1". ("**see illustration #6 ***) 


This piece reveals the physical contents of an Apple][ computer, its state is represented in 
the pattern of data particles printed out and thus made visible. Hohn calls it a self-portrait 
made by the computer and jokingly claims that its self-consciousness is located in those 
memory locations that contain the print program. Of course Hohn is aware of the fact that the 
computer does not know what it is doing when it is dumping its entire memory, but he is interested 
in confronting the audience with the question of the nature of the digital image and how it differs 
from other referential methods. 


One of the titles, "Ceci n'est pas une pomme” ("This is not an Apple"), is a direct 
allusion to the famous painting by the surrealist painter René Magritte, entitled "Ceci n'est pas 
une pipe” ("This is not a pipe"). Magritte’s painting depicts a realistical looking pipe and 
beneath it an inscription with the title. The implied relationship between image and text usually 
would support the assumption that both are incarnations of the same thing; here, this 
interpretation is disturbed by the negation "This is not a pipe”. In discussing the meaning of the 
paradox that we see an image of a pipe, but its title states that it is not a pipe, Michel Foucault 
points to the distinction of referential symbol systems defined as image and text: 


"Two principles, | believe, ruled Western painting from the fifteenth to the twentieth 
century. The first asserts the separation between plastic representation (which implies 
resemblance) and linguistic reference (which excludes it). By resemblance we demonstrate and 
speak across difference: The two systems can neither merge nor intersect. In one way or another, 
subordination is required. Either the text is ruled by the image (as in those paintings where a 
book, an inscription, a letter, or the name of a person are represented); or else the image is ruled 
by the text (as in books where a drawing completes, as if it were merely taking a short cut, the 
message that words are charged to represent).” (Foucault 1983) 


The distinction between image and text, that in Foucaults analysis could only be reconciled 
by subordination of one to the other, is completely blurred when the information is encoded in 
digital format. The abstraction of the data particle accounts for nothing less than the ability to 
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treat all data equally, regardless of its origin, or to interpret data that might have been distilled 
from text as visual information. The consequences of this interchangeability of different symbolic 
representations in their digital form has already started to surface in two major areas of 
application: electronic publishing and artistic collaging. 


3.3.2. Linguistic Symbols as Icons 

The advent of digital media requires the design of new fonts suited to the rasterized format of 
bitmapped CRT display and high resolution hardcopy devices like laser printers. The digital 
typographer Bigelow acknowledges this task by stating that “each technology has its own unique 
way of rendering letterforms." and recommends “to design digital fonts which adhere to the 
principles of readability found in traditional letterform designs, while tuning the features and 
details of the design to the digital medium." (Bigelow 1985) 


Bigelow sees the goal of a good font design to enhance the ease of reading text and 
distinguishes three levels of legibility: readable, legibile, and decipherable. He concludes:"The 
most ‘readable’ text font designs are transparent to the reader. Such fonts are designed to present 
the text in the clearest way possible, while remaining essentially invisible themselves. Thus, the 
best font designs are never noticed as such by the reader. Their qualities are transferred to the 
reading experience.” (Bigelow 1985) 


While Bigelow correctly stresses that the "invisibility" of the fonts generally heips to let 
the reader concentrate on the symbolic message transmitted with them, we want to focus on those 
cases where the "visibility" of the fonts intentionaily carries additional pictorial meaning. Fonts 
have been used as icons particularily in advertising and visual poetry to attract the reader's 
attention and to underline and complement the linguistic message with an image. In the context of 
our discussion it will become clear that both fonts and images share the same abstract notation in 
patterns of data particles. As a consequence the data is readily convertable from one domain into 
the other, text can be expressed in pictorial terms, and text and graphics can be collaged so easily 
with the computer that some experts now speak of digital imagesetting in comparison to 
typesetting with conventional printing technology. 


A letter can be represented by binary data in several distinctly different ways: as a numeric 
code, as an array of pixel data, and as geometric data for control points that in turn can generate an 
outline font with a suitable curve-fitting algorithm. The latter are splines that lend themselves 
very well to efficiently model the intricacies of letter forms. The abstractness of the data particle 
allows encoding letters in a geometric form that is equivalent to the data format other icons are 
represented in. For instance, the French artists Hervé Huitric and Monique Nahas wrote an 
elaborate software package, RODIN, to create mountains, cars, and human bodies; all these 
arbitrarily shaped surfaces are based on B-Splines. (Huitric, Nahas 1983) The malleability of 
the abstract data is further demonstrated in their animated movie "9600 Bauds", when a mountain 
undergoes a metamorphosis into a human head. (*** illustration # 3 ***) 


The graphics language PostScript offers a similar flexibility for the graphical treatment of 
fonts that are represented in spline format. The "Calendar 1985", designed by Cleo Huggins in 
PostScript for Adobe Systems, could well be entitled "The Picture of the Year". It is made entirely 
of transformed fonts that depict seasonal qualities of each month, such as growth activities or the 
celestial passage of the sun in each of the calendar's panels. Moreover,when the panels are 
arranged in a scroll side by side, the entire piece shows the continual changes of a year passing by, 
since the imagery freely flows from month to month. (*** illustration #1 ***) 
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3.3.3. ‘Virtual’ and ‘Real’ Images 

The extent to which the data particle lacks any physicality is most directly experienced by 
those professionals who explicitly dealt with the tangible aspects of their medium; printer's ink, 
steel, wood, glass, etc. A sculptor is particularily concerned with the realities of the medium, 
which have to be crafted into shape with special tools. Artistic content and meaning is embodied in 
the physical structure of the sculpture. This concept of sculpture is completely revised by 
Michael O'Rourke, an artist who works at New York Institute of Technology's Computer Graphics 
Lab: 


"The essence of sculpture, the heart of its matter, lies not so much in sculpture's 
physicality as it does in its conscious composition of three dimensional spaces and volumes. ... | 
suggest, therefore, that, to the extent to which sculpture can be defined, not in terms of its 
physical presence, but in terms of its three dimensional relationship, we can think of sculpture as 
being not necessarily ‘real’ and physical, but also perhaps ‘virtual'- that is, defined, but having no 
physical being." (O'Rourke 1985) 


One outgrowth of the concept of virtuality are O'Rourkes' interactive sculptures. These are 
light sculptures modeled on a real-time computer graphics display that can be experienced by the 
viewer from different vantage points by moving the sculpture, not the viewer. The viewer can 
even freely "enter" the inside of this "virtual" sculpture because it has no physical constrains. In 
a different vein O'Rourke puts his interest in dimensionality to work when he plays with the 
ambiguity that characterizes three dimensional objects when they are projected onto two 
dimensional displays. "The intention is that the viewer will experience a tension between his 
tendency to view it as a two dimensional pattern and as a three dimensional 'thing’." (O'Rourke 
1985) 


The oscillation between three and two dimensional perception is achieved by an imaging 
process that involves complex three dimensional solid modelling as well as image processing in the 
frame buffer plane. First, O'Rourke designs a three dimensional wire-frame object on a real-time 
display by employing boolean operators to cut, intersect and combine geometric primitives. The 
resulting object is split into two parts, one of which is rendered on a frame buffer with smooth 
shading to emphasize its three dimensionality, the other one is simply drawn into the buffer in its 
wire-frame format. O'Rourke then applies various image processing techniques to this second 
image; its lines are blurred, then it is run through filters to enhance the edges, the color of the 
image is inverted and later recombined with the original, etc. At this stage the two images are 
digitally composited, cropping of the background takes place, and to make the three dimensional 
illusion even more "real", a projected shadow of the shaded model is inserted. (*** illustration # 
8.1-8.9) 


The unconventional combination of two dimensional pixel processing and three dimensional 
geometric modelling gives ST. SLUCID its particular perceptional power. The appearance of "real" 
shadow casting things is juxtaposed against the accentuated two dimensionality of the display plane. 
It took a sculptor to discover the potential of the virtuality of the digital data to display 
simultaneously multi-dimensional imagery. O'Rourke used the digital virtual properties also, 
when he first developed the Sequence of images at a lower resolution and then recomputed them at 
2K by 2K for high definiton detail. 


The abstract data describing an image is subject to interpretation in the real world of 
display and hardcopy devices. The image loses its virtuality only after conversion into analog 
format, but remains completely flexible within the digital design process. Depending on purpose, 
requirements, and available hardcopy device, the same image can be rendered at different 
resolutions at will. An instructive example is provided by the titlepage of SHOWPAGE, the 
newsletter of the San Francisco Bay Area Macintosh Users Group. On one page it contains three 
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different incarnations of a virtual image. The title has been printed on a dot-matrix printer, that 
closely resembles the 72 dpi resolution of the Macintosh display. All cube arrangements have been 
printed with a laser printer at 300 dpi, with the exception of the lower left corner which has been 
rendered on a phototypesetter at 1270 dpi. 


3.3.4. Computer Processes on Display 

Large amounts of data are constantly processed by a computer and account for changes of its 
state in fractions of a second. All of these processes proceed invisible to the human eye unless they 
are being presented in an intelligable and comprehendable format. It is clear that a printout of the 
binary values would be totally useless; not only is their quantity prohibitively large, but the 
naked abtractness of the data does not directly indicate the encoded meaning. Display methods are 
needed to mediate between the lowest levels of data perticles and human perception. This problem 
has recently been addressed more vigorously in terms of user-interfacing. The computer now 
has to provide a constantly up-dated account of its inner workings in addition to performing its 
asigned task. Most methods that have been developed to help facilitate this communication link rely 
on the informational density of visual symbols. 


Researchers at Brown University developed BALSA, a real-time Algorithm Simulator and 
Animator, for applications in computer science education, the research in design and analysis of 
algorithms, and advanced debugging and systems programming. A set of representations has been 
developed to allow single stepping through a program, the synchronized display of subroutine 
calls, the construction of trees and graphs to illustrate searching techniques, and color encoded 
visualizations of sorting algorithms. (Brown 1984) 


Whereas BALSA simulates dynamic processes in dynamic displays, a Network Manager 
has been developed by Paul Haeberli at Silicon Graphics to give the user functional control over 
the system's hardware. This Network Manager is running on the IRIS, a graphics workstation that 
features a pipeline of Geometry Engines (Clark 1982). These are VLSI chips that execute four by 
four matrix computations for the real-time display of three dimensional objects. The Network 
Manager enables the user to graphically patch together pathways into data networks and to 
interactively set the values of object's coordinates and the transformational matrices. In effect, 
the user can visually connect data files to be processed by the Geometry Engines and set the matrix 
values inside the VLSI chips. (***illustration #9a**) 


The system offers a visual curve editor to create objects whose data can then be routed toa 
sweep editor that controls how the initial curve will be incrementally transformed to create a 
three dimensional object (i.e. surfaces of revolution, extrusion by translation). This object data 
can then be fed to a transformation matrix to set the global viewing parameters. The user defines 
the hierachical organization of a scene and its object's transformations by simply patching 
together a network. The diagrammatic rules of the Network Manager simulate the paradigm of 
analog synthesizers which were programmed by patching processing modules together with cables. 
At all times there is full interactive control provided over the data generated and transformed. 
Changes applied to any one of the editors are propagated instantaneously throughout the network 
and simultaneously affect the real-time up-date of the display. The user is effectively relieved 
from writing programs, explicitly creating data files, etc. and can operate exclusively in the 
visual domain of the graphics problem to be resolved. 


To summarize the importance of this prototype, | would like to point out, that it manifests 
an attempt to facilitate a visually based dialog between user and machine that far exceeds common 
menu based systems. The user's commands are diagrammatically issued and interpreted as a 
connection of data-driven processing modules. Images serve two different purposes in the Network 
Manager, in one case they represent the command and data structure, and in the other they display 
the results of the executed program. In both instances the computer functions as a mediator 
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between its data particles and visual artifacts. 


4. INTELLIGENT IMAGING 

To this day most computer graphics utilized the machine merely as a mechanistic tool to 
render images or enumerate design possibilites, etc. The Japanese artist and philosopher Hiroshi 
Kawano Calls all of these techniques surface processing, because they result in artifacts that 
just have a perceivable appearance, but no seif-reflective substance. These artifacts are not 
endowed with meaning by the machine processes that generated them. Only the cognitive labor by 
the human operator or viewer can attach meaning to the symbols that are part of the formailistic 
puzzle played by the machine. If these symbols stand for anything in the machine, they stand for 
another layer of codes, they are self-referential. To break this internal loop, Kawano conceived 
the "pragmatic processor DORAEMON" in analogy to the human mind. "A human mind is 
self-reflective, that is, it has a meta-action for object-oriented actions of data processing." 
(Kawano 1982) Consequently, DORAEMON is equipped with a self-reflective meta-part ID and an 
object-reflexive EGO. ID serves as a pragmatic ACTOR and exerts its "intentional will+wish" by 
message-passing to EGO for the actual execution. (Kawano 1982, 1984, 1985) 


Faculties such as understanding and intention are not even rudimentarily cultivated in the 
few existing artificial intelligence imaging systems. Most notable among them is Harold Cohen's 
program AARON that he has conceived, designed and constantly revised since the early 70's. In 
today's jargon the system would be called an expert system that provides “a convincing 
simulation of human performance" in drawing images. Before starting to writing the program 
Cohen analysed the problem space by probing into the basic question: "What is an image?", and 
then rephrases it more accurately: "What would be the minimum condition under which a set of 
marks may function as an image?" (Cohen 1979) In an animated dialog with two children Harold 
Cohen answers these questions and explains how he transferred some of his experiential knowledge 
as a painter into the rules that drive AARON's behavior. (Cohen H, Cohen B, Nii P 1984) In our 
context we are more interested in following Cohen's concise discussion of the "the nature of 
image-mediated transactions” | 


Cohen clearly realizes that his system is only an "evocation generator", which does not 
produce images containing any intended meaning per se, but is "aimed primarily at sustaining an 
illusion" with the simulated dynamics of free-hand drawing. This illusion rests on the clever 
exploitation of AARON's “ability to echo the structure of visual experience, which gives the image 
its plausability. ... In other words, evidence of cognitive process may be substituted for the results 
of an act of cognition. ... Once that is accomplished the transactions which its drawings generate are 
real, not illusory. Like its human counterpart, AARON succeeds in delineating a meaning-space for 
the viewer, and as in any normal transaction not totally prescribed by prior cultural 
arrangements, the viewer provides plausible meanings." (Cohen 1979) 


Cohen accomplished the building of a computer system that creates numerous "free-hana" 
drawings of a particular style independently from human interaction. The manifold components of 
the rule-based drawings exhibit such a well-formed syntactic structure as to suggest meaning to 
the viewer. Moreover, Cohen unfolds with his work the complexity of problems we continue to face 
with regard to the meaningfut production of imagery with computers. 

The real advances of computer graphics technology are still ahead of us, when modes of 
human creative endeavor such as emotion, association and intuition, which are the creative 
ingredients of art, can be successfully simulated by computers. The act of creation itself might 
become automated by learning machines that can self-modify their programs and thus interact 
intelligently and autonomously with the world. Techniques can be adapted from research done in 
artificial intelligence, but the rational logic employed there needs to be revised to model a 
qualitively different kind of logic for artificial creativity. Only then will art making machines 
instill new meaning to the process of bridging mind and matter. 
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